Using inertial sensors to provide smoothed exposure and white balance adjustments for video and photographic applications

ABSTRACT

A first image frame is gathered. The first image frame, a first white balance value, and a first exposure value are stored. Information is sensed from an inertial sensor. A second image frame and intermediate image frames are gathered. A distance traversed by a camera is determined. In response to one or more of: the distance exceeding a predetermined threshold, or a new white balance value significantly differing from a previous white balance value, or a new exposure value significantly differing from a previous exposure value, a second white balance value and a second exposure value are determined for the second image frame. A smoothed white balance value and a smoothed exposure value are determined for the intermediate image frames.

TECHNICAL FIELD

The present invention relates generally to image capture and processing.More particularly, the invention relates to improved systems andtechniques for providing smoothed exposure and white balance adjustmentsfor video and photographic applications.

BACKGROUND

Still-image photography and full-motion video have been continuallyimproved over the years. From Nièpce's photo etchings beginning in 1822,through the development by Nièpce and Daguerre of photography usingsilver compounds, through the development of inexpensive and widelyavailable cameras, through instant photography continuing with moderndigital photography, the ability to record still and moving images hasbecome more and more widespread until it is now nearly universal. Manypeople document every aspect of their lives by gathering anddistributing images of interesting or noteworthy events.

These days, photography is often casual, carried out under widelyvarying lighting conditions, by photographers who are not interested inmaking meticulous camera adjustments. Nonetheless, such photographerswould like their photographs and videos to look attractive and toresemble the subject, particularly in terms of such properties as whitebalance. White balance is a camera setting that adjusts for lighting inorder to make white objects appear white in a photo or video. This ismore difficult than it might seem to be, due to the fact that the lightcast from two different sources may differ substantially in terms ofspectral content and color. Ambient light is very rarely truly white innature. The spectral content of a light source may be referred to ascolor temperature. The light from an incandescent or halogen bulb, forexample, may emphasize the orange and yellow regions of the spectrum,while a fluorescent light may emphasize the blue region of the spectrum.

A proper white balance setting in a camera will prevent a white bedsheet in a photo from appearing to be orange, for example, while thesheet is being illuminated by candlelight. Because it is particularlyimportant that neutral colors (such as white or gray) appear correctly,white balance may also be referred to as color balance or gray balance.White balance is a measure of the relative intensities of each colorcomponent of an image (such as the red, green, and blue additive primarycolors of light). Color constancy is a characteristic of the humanvisual system—the human brain provides a relatively constant colorperception over a significant range of wavelength variation of lightilluminating the subject. Artificial mechanisms for capturing images, onthe other hand, need to incorporate mechanisms for adjustment andcorrection in order to achieve such constancy.

Automatic white balance (AWB) and auto exposure (AE) algorithms employedin the camera imaging pipeline are critical to the color appearance ofdigital pictures and videos. The goal of AWB algorithms is to providethe color constancy feature of the human visual system, wherein theperceived color of an object remains substantially constant underdifferent conditions of illumination. Thus, AWB algorithms have todetermine, from the image content itself, the necessary color correctionto perform in response to the current illumination. For video, the AWBalgorithm is usually employed on a per-frame basis to account for scenechanges. Auto exposure algorithms aim to adjust the camera's exposuretime to minimize overexposed and underexposed areas. This functionalityis needed since cameras are only capable of measuring a limited range ofthe total illumination in the scene (i.e., cameras have limited dynamicrange). To capture bright areas, a shorter exposure time is needed;otherwise, the image will be overexposed or saturated (white). On theother hand, in dark areas a longer exposure time is needed; otherwise,the image will be underexposed or black. The goal of AE algorithms is tofind the optimal exposure for the scene being captured. When capturingvideo, this is typically done using a control loop that analyzesincoming video frames and provides estimates of the best exposure timeto capture subsequent frames.

AWB algorithms attempt to estimate the illumination in an image bymaking some assumptions regarding the distribution of colors in theimage. Then, these AWB algorithms correct the image as if it was takenunder standard illumination. Some of the most well known and widely usedAWB algorithms are the white patch algorithm, which assumes that thebrightest point in the image is white (maximum reflectance); and thegray world algorithm which assumes that the average reflectance of ascene is achromatic. Other AWB algorithms are based upon certainassumptions regarding the content of an image.

AE algorithms typically analyze exposure statistics of an image (e.g.,an intensity histogram) to determine how much the exposure time shouldbe changed in order to obtain “optimal” statistics for the next capturedimage. Different criteria for defining the “optimal” exposure may beused, and those typically follow a heuristic approach. Illustrativeheuristic approaches include having the mean intensity value close to aneutral gray, prioritizing shadow areas, constraining the optimizationto a region pre-set by the user, or prioritizing skin tones.

Conventional AWB and AE algorithms do not provide optimal performance inmany real-world scenarios. These AWB and AE algorithms are executed on aper-frame basis while capturing video sequences. This often leads toundesirable tonal fluctuations and exposure variations in video frames,as is commonly observed in the context of mobile cameras.

SUMMARY OF SOME EMBODIMENTS

According to one embodiment, a method comprises gathering a first imageframe; storing in a memory buffer the first image frame, a first whitebalance value for the first image frame, and a first exposure value forthe first image frame; sensing information from one or more inertialsensors in a camera; gathering a second image frame and gathering one ormore intermediate image frames after the gathering of the first imageframe and prior to the gathering of the second image frame; using thesensed information to determine a distance traversed by the camera fromthe gathering of the first image frame to the gathering of the secondimage frame and, in response to the distance exceeding a predeterminedthreshold, determining a second white balance value and a secondexposure value for the second image frame; and determining, estimating,or calculating a smoothed white balance value and a smoothed exposurevalue for the one or more intermediate image frames.

According to another embodiment, a method comprises gathering a firstimage frame; storing in a memory buffer the first image frame, a firstwhite balance value for the first image frame, and a first exposurevalue for the first image frame; gathering a second image frame andgathering one or more intermediate image frames after the gathering ofthe first image frame and prior to the gathering of the second imageframe; storing in a memory buffer the second image frame, a second whitebalance value for the second image frame, and a second exposure valuefor the second image frame; determining whether or not the first whitebalance value is within a first specified threshold of the second whitebalance value; and determining whether or not the first exposure valueis within a second specified threshold of the second exposure value; andin response to the first white balance value not being within the firstspecified threshold of the second white balance value, or in response tothe first exposure value not being within the second specified thresholdof the second exposure value, then determining, estimating, orcalculating a smoothed white balance value and a smoothed exposure valuefor the one or more intermediate image frames.

According to another embodiment, an apparatus comprises at least oneprocessor, and at least one memory including computer program code, theat least one memory and the computer program code being configured, withthe at least one processor, to cause, at least in part, the apparatus toperform gathering a first image frame; storing in a memory buffer thefirst image frame, a first white balance value for the first imageframe, and a first exposure value for the first image frame; sensinginformation from one or more inertial sensors in a camera; gathering asecond image frame and gathering one or more intermediate image framesafter the gathering of the first image frame and prior to the gatheringof the second image frame; using the sensed information to determine adistance traversed by the camera from the gathering of the first imageframe to the gathering of the second image frame and, in response to thedistance exceeding a predetermined threshold, determining a second whitebalance value and a second exposure value for the second image frame;and determining, estimating, or calculating a smoothed white balancevalue and a smoothed exposure value for the one or more intermediateimage frames.

According to another embodiment, an apparatus comprises at least oneprocessor, and at least one memory including computer program code, theat least one memory and the computer program code being configured, withthe at least one processor, to cause, at least in part, the apparatus toperform gathering a first image frame; storing in a memory buffer a thefirst image frame, a first white balance value for the first imageframe, and a first exposure value for the first image frame; gathering asecond image frame and gathering one or more intermediate image framesafter the gathering of the first image frame and prior to the gatheringof the second image frame; storing in a memory buffer the second imageframe, a second white balance value for the second image frame, and asecond exposure value for the second image frame; determining whether ornot the first white balance value is within a first specified thresholdof the second white balance value; and determining whether or not thefirst exposure value is within a second specified threshold of thesecond exposure value; and in response to the first white balance valuenot being within the first specified threshold of the second whitebalance value, or in response to the first exposure value not beingwithin the second specified threshold of the second exposure value, thendetermining, estimating, or calculating a smoothed white balance valueand a smoothed exposure value for the one or more intermediate imageframes.

According to another embodiment, a computer program product comprisescode stored in a non-transitory computer readable memory for performingat least the following when executed on a computer device: gathering afirst image frame; storing in a memory buffer the first image frame, afirst white balance value for the first image frame, and a firstexposure value for the first image frame; sensing information from oneor more inertial sensors in a camera; gathering a second image frame andgathering one or more intermediate image frames after the gathering ofthe first image frame and prior to the gathering of the second imageframe; using the sensed information to determine a distance traversed bythe camera from the gathering of the first image frame to the gatheringof the second image frame and, in response to the distance exceeding apredetermined threshold, determining a second white balance value and asecond exposure value for the second image frame; and determining,estimating, or calculating a smoothed white balance value and a smoothedexposure value for the one or more intermediate image frames.

According to another embodiment, a computer program product comprisescode stored in a non-transitory computer readable memory for performingat least the following when executed on a computer device: gathering afirst image frame; storing in a memory buffer the first image frame, afirst white balance value for the first image frame, and a firstexposure value for the first image frame; gathering a second image frameand gathering one or more intermediate image frames after the gatheringof the first image frame and prior to the gathering of the second imageframe; storing in a memory buffer the second image frame, a second whitebalance value for the second image frame, and a second exposure valuefor the second image frame; determining whether or not the first whitebalance value is within a first specified threshold of the second whitebalance value; and determining whether or not the first exposure valueis within a second specified threshold of the second exposure value; andin response to the first white balance value not being within the firstspecified threshold of the second white balance value, or in response tothe first exposure value not being within the second specified thresholdof the second exposure value, then determining, estimating, orcalculating a smoothed white balance value and a smoothed exposure valuefor the one or more intermediate image frames.

According to another embodiment, an apparatus comprises means forperforming the steps of one of the above methods.

According to another embodiment, a computer program product includes oneor more sequences of one or more instructions which, when executed byone or more processors, cause an apparatus to at least perform the stepsof one of the above methods.

Still other aspects, features, and advantages of the invention arereadily apparent from the following detailed description, simply byillustrating a number of particular embodiments and implementations,including the best mode contemplated for carrying out the invention. Theinvention is also capable of other and different embodiments, and itsseveral details can be modified in various obvious respects, all withoutdeparting from the spirit and scope of the invention. Accordingly, thedrawings and description are to be regarded as illustrative in nature,and not as restrictive.

BRIEF DESCRIPTION OF THE FIGURES

Various embodiments are illustrated by way of example, and not by way oflimitation, in the figures of the accompanying drawings in which:

FIG. 1 is a flowchart of a first exemplary set of processes forproviding smoothed exposure and white balance adjustments for video andphotographic applications according to one set of embodiments.

FIG. 2 is a flowchart of a second exemplary set of processes forproviding smoothed exposure and white balance adjustments for video andphotographic applications according to one set of embodiments.

FIG. 3 is a flowchart of a third exemplary set of processes forproviding smoothed exposure and white balance adjustments for video andphotographic applications according to one set of embodiments.

FIG. 4 is a diagram of a first device that can be used to implement anembodiment of the invention.

FIG. 5 is an alternate view of the device shown in FIG. 4.

FIG. 6 is a diagram of a second device that can be used to implement anembodiment of the invention.

DESCRIPTION OF SOME EMBODIMENTS

Examples of methods, apparatuses, and computer program products aredisclosed for providing smoothed exposure and white balance adjustmentsfor video and photographic applications. In the following description,for the purposes of explanation, numerous specific details are set forthin order to provide a thorough understanding of the embodiments of theinvention. It is apparent, however, to one skilled in the art that theembodiments of the invention may be practiced without these specificdetails or with an equivalent arrangement. In other instances,well-known structures and devices are shown in block diagram form inorder to avoid unnecessarily obscuring the embodiments of the invention.

According to a set of illustrative embodiments disclosed herein, anadaptation or calculation rate of auto white balance and auto exposurealgorithms are changed or adjusted, based on inertial sensors (such asaccelerometers and gyroscopes) in the camera to reduce undesirable tonalfluctuation. The inertial sensor values are used to calculate anaccumulative distance the camera has traveled, and only when thisdistance reaches a certain threshold new values of white balance andexposure are adopted or recalculated for the incoming frame. This cancreate a jump in the exposure and white balance settings at this frame.To smooth it out, the most recent captured frame or frames are stored ina memory buffer and the white balance and exposure settings areinterpolated for the frame or frames between the frame when the exposureand white balance were last changed and the current frame. In contrastwith the prior art, which changes exposure and white balance parameterson a per-frame basis, our invention produces more stable videosregarding tonal settings.

FIG. 1 is a flowchart of a first exemplary set of processes forproviding smoothed exposure and white balance adjustments for video andphotographic applications according to one set of embodiments. Theoperational sequence commences at block 101 where a first image frame isgathered. Next, at block 103, the first image frame, a first whitebalance value for the first image frame, and a first exposure value forthe first image frame, are all stored in a memory buffer. Information issensed from one or more inertial sensors in a camera (block 105). Asecond image frame is gathered, and one or more intermediate imageframes are gathered after the gathering of the first image frame butprior to the gathering of the second image frame (block 107).Illustratively, the first image frame, the second image frame, and theone or more intermediate image frames are stored in the buffer. In somecases, it is not necessary to store exposure and white balance valuesfor every frame of the one or more intermediate frames, as these valuesmay be interpolated based upon the first image frame and the secondimage frame, or based upon a previous image frame and a successive imageframe of the one or more intermediate frames. The sensed information isused to determine a distance traversed by the camera from the gatheringof the first image frame to the gathering of the second image frame(block 109).

In response to the distance exceeding a predetermined threshold, asecond white balance value for the second image frame and a secondexposure value for the second image frame are determined (block 111). Asmoothed white balance value for the one or more intermediate imageframes and a smoothed exposure value for the one or more intermediateimage frames are determined, estimated, or calculated by interpolatingthe first white balance value and the second white balance value, and byinterpolating the first exposure value and the second exposure value, toreduce a white balance fluctuation between the first white balance valueand the second white balance value for the one or more intermediateimage frames, and to reduce an exposure fluctuation between the firstexposure value and the second exposure value for the one or moreintermediate image frames (block 113).

Conventional auto-exposure and auto white balance algorithms areexecuted on a per-frame basis while capturing video sequences. Thisoften leads to undesirable fluctuations in color tone and exposure invideo frames, as is commonly observed with still images and videos thatare gathered using mobile cameras. The procedures of FIGS. 1-3 changesthe frequency for computing values for automatic exposure and automaticwhite balance during video capture to adapt to camera movement. Insteadof computing values for automatic exposure and automatic white balancefor every captured frame, one or more inertial sensors are provided(such as gyroscopes or accelerometers) to measure an approximatemovement or motion of the camera, and set the frequency of evaluation ofautomatic exposure and automatic white balance according to an amount ofmeasured movement or motion. If it is detected that the camera is still,the frequency is lowered so that the automatic white balance andautomatic exposure values remain fixed for a short time interval. Thisscenario assumes that illumination and scene changes are unlikely duringa short time interval. If motion is detected, the frequency is increasedso that exposure and white balance are computed more often to deal withpotential scene changes due to the camera movement. Since automaticwhite balance and automatic exposure will be calculated and changed lessoften, more sophisticated algorithms for computing white balance andautomatic exposure may, but need not, be employed to reduce color tonalfluctuations and exposure variations.

FIG. 2 is a flowchart of a second exemplary set of processes forproviding smoothed exposure and white balance adjustments for video andphotographic applications according to one set of embodiments. First,initialize a motion accumulator to zero (block 201). Then, capture afirst video frame at block 202. At block 203, calculate a cameramovement (amount and direction of movement) during the frame time(1/frame rate) based on an inertial sensor 204. Also at block 203, usean auto white balance (AWB) algorithm 209 to calculate a new whitebalance setting using the first video frame as input, and use an autoexposure (AE) algorithm 211 to calculate a new exposure setting usingthe first video frame as input.

At block 205, the camera movement that was calculated at block 203 isadded to the motion accumulator to calculate the total distance thecamera has traveled. At block 207, if the distance traveled reaches apredetermined threshold, then the new values of white balance andexposure computed at block 203 are applied at block 215, i.e., theyreplace a formerly existing value of white balance and a formerlyexisting value of exposure, so that the capture of new (subsequent)frames (block 202) from now on uses these new values. If the buffer 222is full (block 226), then remove the oldest frame from the buffer 222and send the oldest frame to a video storage pipeline (block 228). Thecurrent frame is then added to the buffer (block 224). After this frameis added to the buffer 222, a smoothing algorithm (block 220), to bedescribed in greater detail hereinafter, is executed. The operationalsequence then loops back to block 201 where the motion accumulator isreset to zero.

Also at block 207, a test is performed to ascertain whether or not thevalue of white balance calculated at block 209, or the value of exposurecalculated at block 211, or both the value of white balance calculatedat block 209 and the value of exposure calculated at block 211, areoutside of a specified threshold with reference to corresponding valuesof white balance or exposure from one or more previous frames (i.e., dida significant change in white balance or exposure occur?). If one orboth of the value of white balance and the value of exposure are outsideof the specified threshold, then this change is most likely due toillumination changes, and the new computed white balance and exposurevalues are applied as was described previously in connection with block215. If the buffer 222 is full, the oldest frame in the buffer is pushedout and sent to the video processing pipeline for proper storage (block228). The current frame is added to the buffer 222 (block 224), and thesmoothing algorithm is executed (block 220). The motion accumulator isthen reset to zero (block 201). Otherwise (if both the value of whitebalance and the value of exposure fall within the specified thresholdand the distance traveled by the camera is below a predetermined value),the white balance and exposure settings are kept unchanged (block 213);if the buffer 222 is full, the oldest frame is pushed out and sent tothe video processing pipeline for proper storage (block 228). Thecurrent frame is added to the buffer (block 224), and the procedurecontinues by capturing a new frame (block 202).

Pursuant to the operational sequence of FIG. 2, the new computedexposure and white balance values are only applied once the camera ismoved by a certain amount or the estimated white balance or exposuresettings significantly differ from their previous estimates. When thoseconditions do not hold, the white balance and exposure settings are keptunchanged. Therefore, the procedure of FIG. 2 reduces small undesirablefluctuations, but still allows the exposure and white balance to adaptwhen there are significant changes in illumination. The jumps in whitebalance and exposure settings between subsequent groups of frames takenwith different settings are smoothed out by keeping a buffer with themost recent captured frames and applying a smoothing algorithm.

FIG. 3 is a flowchart of a third exemplary set of processes forproviding smoothed exposure and white balance adjustments for video andphotographic applications according to one set of embodiments. Thealternative implementation of FIG. 3 reduces the computationalcomplexity of a video pipeline by only recalculating the white balanceand exposure when the camera has moved a certain distance. Thisimplementation has the characteristic that it cannot detect significantillumination changes. However, it may be used in conjunction with anillumination sensor, or it may be used for short video clips wheresignificant illumination changes, such as turning a light switch, areunlikely.

The operational sequence of FIG. 3 commences at block 301 where a motionaccumulator is initialized to zero. At block 302, a new video frame iscaptured. Then, at block 303, a camera movement (amount and direction ofmovement) is calculated during the frame time (1/frame rate) based onthe inertial sensor 204. The camera movement computed at block 303 isadded to the accumulator at block 305 to calculate the total distancethe camera has traveled. If the distance traveled reaches or exceeds acertain threshold (block 307), then the automatic white balance (AWB)algorithm 209 and the auto exposure (AE) algorithm 211 are triggered(block 312) to calculate new values for white balance and exposuresettings. The new values for white balance and exposure settings areapplied (block 315) from that frame forward. Otherwise, the negativebranch from block 307 leads to block 313 where the old (existing) valuesof the white balance and exposure are kept. In either case, if thebuffer 222 is full (block 326), the oldest frame in the buffer is pushedout and sent to the video processing pipeline for proper storage (block328). The current frame is then added to the buffer 222 (block 324).

If the operational sequence of FIG. 3 progressed through block 313, thenprocessing continues by capturing a new frame (block 302) with old(existing) settings. If the operational sequence of FIG. 3 progressedthrough block 315, capturing is done with the new settings, thesmoothing algorithm is executed (block 320), and the program loops backto block 301 where the accumulator is reset to zero.

For the smoothing algorithm (block 320, FIG. 3 and block 220, FIG. 2),one approach is to obtain the white balance and exposure values for theframe when the exposure and white balance were last changed and thefirst frame taken with the new settings, and interpolate the whitebalance and ISO settings for the intermediate frames in between (as theframes in the buffer have already been captured, their exposure timescannot be changed any more, but their ISO settings can be modified inorder to compensate for it). This makes the assumption that those valuesare global settings in the frame, which is not always true. Anotherapproach is to use the algorithm proposed in the paper “TonalStabilization of Video” by Farbman et al., SIGGRAPH 2011, by consideringthe frame when the exposure and white balance were last changed, and thefirst frame taken with the new settings, as anchor frames.Alternatively, other similar approaches could be used to smooth out thetransition. Using this type of buffer is in sync with next-generationcamera architectures, where a circular buffer is planned forapplications such as Zero Shutter Lag.

Referring to FIG. 4, there is shown a front view of an illustrativeelectronic device 10 incorporating features of the invention. Althoughthe invention will be described with reference to the exemplaryembodiments shown in the drawings, it should be understood that theinvention can be embodied in many alternate forms of embodiments. Inaddition, any suitable size, shape or type of elements or materialscould be used.

According to one example of the invention, the device 10 is amulti-function portable electronic device. However, in alternateembodiments, features of the various embodiments of the invention couldbe used in any suitable type of portable electronic device such as amobile phone, a gaming device, a music player, a notebook computer, acamera device, or a personal digital assistant, for example. Inaddition, as is known in the art, the device 10 can include multiplefeatures or applications such as a camera, a music player, a gameplayer, or an Internet browser, for example. The device 10 generallycomprises a housing 12, a transmitter 14, a receiver 16, an antenna 18(connected to the transmitter 14 and the receiver 16), electroniccircuitry 20, such as a controller (which could include a processor [orcentral processing unit (CPU)], for example) 22 and a memory 24 forexample, within the housing 12, a user input region 26, a display 28, aninertial sensor 204 such as an accelerometer or gyroscope, and a memorybuffer 222. It should be understood that although the user input region26 is illustrated as a keypad, various exemplary embodiments of theinvention may include touch-screen technology at the user input region26. The display 28 could also form a user input section, such as a touchscreen. It should be noted that in alternate embodiments, the device 10may have any suitable type of features as known in the art.Additionally, all of these components within the device 10 are normallypowered by a portable power supply such as a battery 30.

Referring now also to FIG. 5, the electronic device 10 further comprisesa camera 32 which is shown as being rearward facing (for example forcapturing images and/or video for local storage) but may alternativelyor additionally be forward facing (for example for video calls). Thecamera 32 may be controlled by a shutter actuator 34 and optionally by azoom actuator 36. However, any suitable camera control functions and/orcamera user inputs may be provided.

The electronic device 10 further comprises a camera exposure system 38.According to some embodiments of the invention, the camera exposuresystem 38 is configured to perform the AE algorithm 211 (FIGS. 2 and 3)and the AWB algorithm 209. For example, the choice of exposure timedirectly affects the appearance of the photograph taken. The exposuretime (or shutter speed) determines the amount of time during which thesensor collects light while capturing a picture. This makes the imagebrighter or darker. Also, in photography for scenes with motion, a shortexposure time “freezes” objects in motion; on the other hand, longerexposure times in images provides a ‘blur’ due to the moving objects inthe scene (also known as “motion blur”, which can be used to depictmotion, for example).

Referring now also to FIG. 6, the device 10 generally comprises acontroller 22 such as a microprocessor for example. The electroniccircuitry includes a memory 24 coupled to the controller 22, such as ona printed circuit board for example. The memory 24 could includemultiple memories including removable memory modules and one or morebuffers, for example. Alternatively or additionally, a separate buffer222 may be provided. An inertial sensor 204 may comprise one or moreaccelerometers or gyroscopes. The device has applications 25, such assoftware, which the user can use. The applications can include, forexample, a telephone application, an Internet browsing application, agame playing application, a digital camera application, a map/gpsapplication, and so forth. These are only some examples and should notbe considered as limiting. One or more user inputs 26 are coupled to thecontroller 22 and one or more displays 28 are coupled to the controller22. The camera exposure system 38 is also coupled to the controller 22.

While the invention has been described in connection with a number ofembodiments and implementations, the invention is not so limited butcovers various obvious modifications and equivalent arrangements, whichfall within the purview of the appended claims. Although features of theinvention are expressed in certain combinations among the claims, it iscontemplated that these features can be arranged in any combination andorder.

Operations of the flowcharts of FIGS. 1-3 and combinations of operationsin the flowcharts may be implemented by various means, such as hardware,firmware, processor, circuitry and/or other device associated withexecution of software including one or more computer programinstructions. For example, one or more of the procedures described invarious embodiments may be embodied by computer program instructions. Inan example embodiment, the computer program instructions, which embodythe procedures, described in various embodiments may be stored by atleast one memory device of an apparatus and executed by at least oneprocessor in the apparatus. Any such computer program instructions maybe loaded onto a computer or other programmable apparatus (for example,hardware) to produce a machine, such that the resulting computer orother programmable apparatus embody means for implementing theoperations specified in the flowcharts. These computer programinstructions may also be stored in a computer-readable storage memory(as opposed to a transmission medium such as a carrier wave orelectromagnetic signal) that may direct a computer or other programmableapparatus to function in a particular manner, such that the instructionsstored in the computer-readable memory produce an article ofmanufacture, the execution of which implements the operations specifiedin the flowchart. The computer program instructions may also be loadedonto a computer or other programmable apparatus to cause a series ofoperations to be performed on the computer or other programmableapparatus to produce a computer-implemented process such that theinstructions, which execute on the computer or other programmableapparatus, provide operations for implementing the operations in theflowchart. The operations of the methods of FIGS. 1-3 are described withhelp of apparatus 10 (FIGS. 4-6). However, the operations of the methodscan be described and/or practiced by using any other apparatus.

Various embodiments described above may be implemented in software,hardware, application logic or a combination of software, hardware andapplication logic. The software, application logic and/or hardware mayreside on at least one memory, at least one processor, an apparatus or,a computer program product. In an example embodiment, the applicationlogic, software or an instruction set is maintained on any one ofvarious conventional computer-readable media. In the context of thisdocument, a “computer-readable medium” may be any media or means thatcan contain, store, communicate, propagate or transport the instructionsfor use by or in connection with an instruction execution system,apparatus, or device, such as a computer, with one example of anapparatus described and depicted in FIGS. 4-6. A computer-readablemedium may comprise a computer-readable storage medium that may be anymedia or means that can contain or store the instructions for use by orin connection with an instruction execution system, apparatus, ordevice, such as a computer.

If desired, the different functions discussed herein may be performed ina different order and/or concurrently with other. Furthermore, ifdesired, one or more of the above-described functions may be optional ormay be combined.

Although various aspects of the embodiments are set out in theindependent claims, other aspects comprise other combinations offeatures from the described embodiments and/or the dependent claims withthe features of the independent claims, and not solely the combinationsexplicitly set out in the claims.

It is also noted herein that while the above describes exampleembodiments, these descriptions should not be viewed in a limitingsense. Rather, there are several variations and modifications, which maybe made without departing from the scope of the present disclosure as,defined in the appended claims.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as an apparatus, method, or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit”, “module”, or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer-readablemedium(s) having computer-readable program code embodied thereon.

Any combination of one or more computer-readable medium(s) may beutilized. The computer-readable medium may be a computer-readable signalmedium or a computer-readable storage medium. A computer-readablestorage medium does not include propagating signals and may be, forexample, but not limited to, an electronic, magnetic, optical,electromagnetic, infrared, or semiconductor system, apparatus, ordevice, or any suitable combination of the foregoing. More specificexamples (a non-exhaustive list) of the computer-readable storage mediumwould include the following: an electrical connection having one or morewires, a portable computer diskette, a hard disk, a random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), an optical fiber, a portable compactdisc read-only memory (CD-ROM), an optical storage device, a magneticstorage device, or any suitable combination of the foregoing. In thecontext of this document, a computer readable storage medium may be anytangible medium that can contain, or store a program for use by or inconnection with an instruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signalwith computer-readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer-readable signal medium may be any computer-readable medium thatis not a computer-readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer-readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, and so forth, or any suitablecombination of the foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object-oriented programminglanguage, such as Java, Smalltalk, C++ or the like and conventionalprocedural programming languages, such as the “C” programming languageor similar programming languages. The program code may execute entirelyon the user's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer, for example, through the Internet using an Internet ServiceProvider.

Aspects of the present invention are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable-medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce acomputer-implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of anyand all means or step plus function elements in the claims below areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of various embodiments has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The embodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

Various modifications and adaptations may become apparent to thoseskilled in the relevant arts in view of the foregoing description, whenread in conjunction with the accompanying drawings. However, any and allmodifications of the teachings of this disclosure will still fall withinthe scope of the non-limiting embodiments of this invention.

Although described in the context of particular embodiments, it will beapparent to those skilled in the art that a number of modifications andvarious changes to these teachings may occur. Thus, while the inventionhas been particularly shown and described with respect to one or moreembodiments thereof, it will be understood by those skilled in the artthat certain modifications or changes may be made therein withoutdeparting from the scope of the invention as set forth above, or fromthe scope of the claims to follow.

We claim:
 1. A method comprising: gathering a first image frame; storingin a memory buffer the first image frame, a first white balance valuefor the first image frame, and a first exposure value for the firstimage frame; sensing information from one or more inertial sensors in acamera; gathering a second image frame and gathering one or moreintermediate image frames after the gathering of the first image frameand prior to the gathering of the second image frame; using the sensedinformation from the one or more inertial sensors to determine adistance traversed by the camera from the gathering of the first imageframe to the gathering of the second image frame and, in response to thedistance exceeding a predetermined threshold, determining a second whitebalance value and a second exposure value for the second image frame;and determining, estimating, or calculating a smoothed white balancevalue and a smoothed exposure value for the one or more intermediateimage frames; else in response to the distance not exceeding thepredetermined threshold, utilizing the first white balance value and thefirst exposure value for the one or more intermediate image frames. 2.The method of claim 1 wherein the determining, estimating, orcalculating is performed by interpolating the first white balance valueand the second white balance value, and by interpolating the firstexposure value and the second exposure value.
 3. The method of claim 2wherein the interpolating of the first white balance value and thesecond white balance value is performed so as to reduce a white balancefluctuation between the first white balance value and the second whitebalance value for the one or more intermediate image frames.
 4. Themethod of claim 2 wherein the interpolating of the first exposure valueand the second interpolation value is performed so as to reduce anexposure fluctuation between the first exposure value and the secondexposure value for the one or more intermediate image frames.
 5. Themethod of claim 1 further comprising determining whether or not thefirst white balance value is within a specified threshold of the secondwhite balance value and, in response to the first white balance valuenot being within the specified threshold of the second white balancevalue, determining the second white balance value and the secondexposure value for the second image frame.
 6. The method of claim 1further comprising determining whether or not the first exposure valueis within a specified threshold of the second exposure value and, inresponse to the first exposure value not being within the specifiedthreshold of the second exposure value, determining the second whitebalance value and the second exposure value for the second image frame.7. The method of claim 1 further comprising using the sensed informationto update a motion accumulator specifying a total distance traversed bythe camera from the gathering of the first image frame.
 8. A methodcomprising: gathering a first image frame; storing in a memory bufferthe first image frame, a first white balance value for the first imageframe, and a first exposure value for the first image frame; gathering asecond image frame and gathering one or more intermediate image framesafter the gathering of the first image frame and prior to the gatheringof the second image frame; storing in a memory buffer the second imageframe, a second white balance value for the second image frame, and asecond exposure value for the second image frame; determining whether ornot the first white balance value is within a first specified thresholdof the second white balance value; and determining whether or not thefirst exposure value is within a second specified threshold of thesecond exposure value; and in response to the first white balance valuenot being within the first specified threshold of the second whitebalance value, or in response to the first exposure value not beingwithin the second specified threshold of the second exposure value, thendetermining, estimating, or calculating a smoothed white balance valueand a smoothed exposure value for the one or more intermediate imageframes; else in response to the first white balance value being withinthe first specified threshold of the second white balance value and thefirst exposure value being within the second specified threshold of thesecond exposure value, then utilizing the first white balance value andthe first exposure value for the one or more intermediate image frames.9. The method of claim 8 wherein the determining, estimating, orcalculating is performed by interpolating the first white balance valueand the second white balance value, and by interpolating the firstexposure value and the second exposure value.
 10. The method of claim 9wherein the interpolating of the first white balance value and thesecond white balance value is performed so as to reduce a white balancefluctuation between the first white balance value and the second whitebalance value for the one or more intermediate image frames.
 11. Themethod of claim 9 wherein the interpolating of the first exposure valueand the second interpolation value is performed so as to reduce anexposure fluctuation between the first exposure value and the secondexposure value for the one or more intermediate image frames.
 12. Anapparatus comprising at least one processor, and at least one memoryincluding computer program code, the at least one memory and thecomputer program code configured, with the at least one processor, tocause, at least in part, the apparatus to perform: gathering a firstimage frame; storing in a memory buffer the first image frame, a firstwhite balance value for the first image frame, and a first exposurevalue for the first image frame; sensing information from one or moreinertial sensors in a camera; gathering a second image frame andgathering one or more intermediate image frames after the gathering ofthe first image frame and prior to the gathering of the second imageframe; using the sensed information from the one or more inertialsensors to determine a distance traversed by the camera from thegathering of the first image frame to the gathering of the second imageframe and, in response to the distance exceeding a predeterminedthreshold, determining a second white balance value and a secondexposure value for the second image frame; and determining, estimating,or calculating a smoothed white balance value and a smoothed exposurevalue for the one or more intermediate image frames; else in response tothe distance not exceeding the predetermined threshold, utilizing thefirst white balance value and the first exposure value for the one ormore intermediate image frames.
 13. The apparatus of claim 12, whereinthe apparatus performs the determining, estimating, or calculating byinterpolating the first white balance value and the second white balancevalue, and by interpolating the first exposure value and the secondexposure value.
 14. The apparatus of claim 13, wherein the apparatusperforms the interpolating of the first white balance value and thesecond white balance value so as to reduce a white balance fluctuationbetween the first white balance value and the second white balance valuefor the one or more intermediate image frames.
 15. The apparatus ofclaim 13, wherein the apparatus performs the interpolating of the firstexposure value and the second interpolation value so as to reduce anexposure fluctuation between the first exposure value and the secondexposure value for the one or more intermediate image frames.
 16. Theapparatus of claim 12, wherein the apparatus determines whether or notthe first white balance value is within a specified threshold of thesecond white balance value and, in response to the first white balancevalue not being within the specified threshold of the second whitebalance value, the apparatus determines the second white balance valueand the second exposure value for the second image frame.
 17. Theapparatus of claim 12, wherein the apparatus determines whether or notthe first exposure value is within a specified threshold of the secondexposure value and, in response to the first exposure value not beingwithin the specified threshold of the second exposure value, theapparatus determines the second white balance value and the secondexposure value for the second image frame.
 18. The apparatus of claim12, wherein the apparatus uses the sensed information to update a motionaccumulator specifying a total distance traversed by the camera from thegathering of the first image frame.
 19. An apparatus comprising at leastone processor, and at least one memory including computer program code,the at least one memory and the computer program code configured, withthe at least one processor, to cause, at least in part, the apparatus toperform: gathering a first image frame; storing in a memory buffer a thefirst image frame, a first white balance value for the first imageframe, and a first exposure value for the first image frame; gathering asecond image frame and gathering one or more intermediate image framesafter the gathering of the first image frame and prior to the gatheringof the second image frame; storing in a memory buffer the second imageframe, a second white balance value for the second image frame, and asecond exposure value for the second image frame; determining whether ornot the first white balance value is within a first specified thresholdof the second white balance value; and determining whether or not thefirst exposure value is within a second specified threshold of thesecond exposure value; in response to the first white balance value notbeing within the first specified threshold of the second white balancevalue, or in response to the first exposure value not being within thesecond specified threshold of the second exposure value, thendetermining, estimating, or calculating a smoothed white balance valueand a smoothed exposure value for the one or more intermediate imageframes; else in response to the first white balance value being withinthe first specified threshold of the second white balance value and thefirst exposure value being within the second specified threshold of thesecond exposure value, then utilizing the first white balance value andthe first exposure value for the one or more intermediate image frames.20. The apparatus of claim 19, wherein the apparatus performs thedetermining, estimating, or calculating by interpolating the first whitebalance value and the second white balance value, and by interpolatingthe first exposure value and the second exposure value.
 21. Theapparatus of claim 20, wherein the apparatus performs the interpolatingof the first white balance value and the second white balance value soas to reduce a white balance fluctuation between the first white balancevalue and the second white balance value for the one or moreintermediate image frames.
 22. The apparatus of claim 20, wherein theinterpolating of the first exposure value and the second interpolationvalue is performed so as to reduce an exposure fluctuation between thefirst exposure value and the second exposure value for the one or moreintermediate image frames.
 23. A computer program product comprisingcode stored in a non-transitory computer readable memory for performingat least the following when executed on a computer device: gathering afirst image frame; storing in a memory buffer the first image frame, afirst white balance value for the first image frame, and a firstexposure value for the first image frame; sensing information from oneor more inertial sensors in a camera; gathering a second image frame andgathering one or more intermediate image frames after the gathering ofthe first image frame and prior to the gathering of the second imageframe; using the sensed information from the one or more inertialsensors to determine a distance traversed by the camera from thegathering of the first image frame to the gathering of the second imageframe and, in response to the distance exceeding a predeterminedthreshold, determining a second white balance value and a secondexposure value for the second image frame; and determining, estimating,or calculating a smoothed white balance value and a smoothed exposurevalue for the one or more intermediate image frames; else in response tothe distance not exceeding the predetermined threshold, utilizing thefirst white balance value and the first exposure value for the one ormore intermediate image frames.
 24. A computer program productcomprising code stored in a non-transitory computer readable memory forperforming at least the following when executed on a computer device:gathering a first image frame; storing in a memory buffer the firstimage frame, a first white balance value for the first image frame, anda first exposure value for the first image frame; gathering a secondimage frame and gathering one or more intermediate image frames afterthe gathering of the first image frame and prior to the gathering of thesecond image frame; storing in a memory buffer the second image frame, asecond white balance value for the second image frame, and a secondexposure value for the second image frame; determining whether or notthe first white balance value is within a first specified threshold ofthe second white balance value; and determining whether or not the firstexposure value is within a second specified threshold of the secondexposure value; and in response to the first white balance value notbeing within the first specified threshold of the second white balancevalue, or in response to the first exposure value not being within thesecond specified threshold of the second exposure value, thendetermining, estimating, or calculating a smoothed white balance valueand a smoothed exposure value for the one or more intermediate imageframes; else in response to the first white balance value being withinthe first specified threshold of the second white balance value and thefirst exposure value being within the second specified threshold of thesecond exposure value, then utilizing the first white balance value andthe first exposure value for the one or more intermediate image frames.